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Today’s  Speaker 


Linda  Northrop  is  director  of  the  Research, 

Technology,  and  Systems  Solution  Program  at  the 
Software  Engineering  Institute  where  she  leads  the 
work  in  architecture-centric  engineering,  software 
product  lines,  systems  of  systems,  and  ultra-large- 
scale  systems. 

She  is  coauthor  of  Software  Product  Lines:  Practices 
and  Patterns.  She  recently  led  a  year  long  study 
including  leaders  in  the  software  community  to  define 
technical  and  social  challenges  to  the  creation  of  ultra- 
large-scale  systems  that  will  evolve  in  the  next 
generation.  The  group  published  the  study  report, 
Ultra-Large-Scale  Systems:  The  Software  Challenge  of 
the  Future  (ISBN  0-9786956-0-7). 


Before  joining  the  SEI,  she  was  associated  with  both 
the  United  States  Air  Force  Academy  and  the  State 
University  of  New  York  as  professor  of  computer 
science,  and  with  both  Eastman  Kodak  and  IBM  as  a 
software  engineer. 
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Polling  Question  1 


How  did  you  hear  about  this  webinar? 

1.  Email  invitation  from  the  SEI 

2.  SEI  Website 

3.  Website  with  webinar  calendar  (ie  www.webinar-directory.com) 

4.  Social  Media  site  (  Linkedln,  Twitter) 

5.  SEI  Member  Bulletin 
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Climate  change  and  the  environment 
Powering  our  civilization 
Disease,  epidemics,  and  health  care 
Livable  megacities 
Safety  and  security 
Transportation 
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Societal  Problems 
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Enormous  web  service  and 
computing  infrastructure 

Supply  chain  systems 

Software-based  engineering 
systems 
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Trend  Toward  Increasing  Scale  -  2 


Healthcare  Infrastructure 
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Increasing  Scale  In  Military  Systems 


Increasingly  Complex  Systems 

•  ultra-large,  network-centric,  real-time, 
cyber-physical-social  systems 

—  thousands  of  platforms,  sensors,  decision 
nodes,  weapons,  and  warfighters 

—  connected  through  heterogeneous  wired 
and  wireless  networks 


Goal:  Information  Superiority 

•  Transient  and  enduring  resource 
constraints  and  failures 

•  Continuous  adaptation 

—  changes  in  mission  requirements 

—  changes  in  operating  environments 

—  changes  in  force  structure 

—  perpetual  systems’  evolution 

—  addition  of  new  systems 

•  Sustainable  -  legally,  technically,  politically 
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The  Challenge 


“Our  soldiers  depend  on  software  and  will 
depend  more  on  software  in  the  future. 

The  Army’s  success  depends  on  software 
and  the  software  industry. 

We  need  better  tools  to  meet  future 
challenges,  and  neither  industry  nor 
government  is  working  on  how  to  do  things 
light-years  faster  and  cheaper. 

How  can  future  systems,  which  are  likely  to 
be  a  billion  lines  of  code,  be  built  reliably 
if  we  can’t  even  get  today’s  systems  right?” 

—  Asst  Sec  Army  Claude  Bolton 
August  16,  2005 


The  Honorable 
Claude  M.  Bolton,  Jr. 
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Ultra-Large-Scale  (ULS)  Systems  Study 


Gather  leading  experts  to  study: 

•  characteristics  of  ULS  systems 

•  challenges  and  breakthroughs  required 

•  promising  research  and  approaches 

Intended  outcomes: 

•  ULS  System  Research  Agenda 

•  program  proposal 

•  collaborative  research  network 


About  the  Effort 

Funded  by  the  Army  (ASA  ALT) 

Staffing:  9  member  SEI  team 

13  member  expert  panel 

Duration:  one  year  (04/05  -  05/06) 
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ULS  Systems  Research  Agenda 


Describes 

•  the  characteristics  of  ULS 
systems 

•  the  associated  challenges 

•  promising  research  areas 
and  topics 

Is  based  on  a  new  perspective 
needed  to  address  the 
problems  associated  with 
ultra-large-scale  systems. 
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Polling  Question  2 


Prior  to  this  webinar  have  you  heard  about  this  study? 


1 )  Heard  about  it  but  haven’t  read  the  report. 

2)  Read  the  report. 

3)  Had  not  heard  about  it. 
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Instead:  A  Different  Kind  of  Study 


It  presents  an  overall  research  agenda  --  not  just  for  new  tools  or  a 
new  software  method  or  modest  improvements  in  today’s  approaches. 

It  is  based  on  the  challenges  associated  with  ultra-large  scale. 

It  focuses  on  the  future. 

It  involves  an  interdisciplinary  base. 

It  takes  a  fresh  perspective  on  the  development,  deployment, 
operation,  and  evolution  of  software-intensive  systems. 


\ 

Germs  of  these  ideas  are  present  today  in  small  research  pockets; 
these  efforts  are  currently  too  small  to  have  much  impact  on  next- 
generation  ULS  systems. 
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What  Is  an  Ultra-Large-Scale  (ULS)  System? 


A  ULS  System  has  unprecedented  scale  in  some  of  these  dimensions: 

•  lines  of  code 

•  amount  of  data  stored,  accessed,  manipulated,  and  refined 

•  number  of  connections  and  interdependencies 

•  number  of  hardware  elements 

•  number  of  computational  elements 

•  number  of  system  purposes  and  user  perception  of  these  purposes 

•  number  of  routine  processes,  interactions,  and  “emergent  behaviors” 

•  number  of  (overlapping)  policy  domains  and  enforceable  mechanisms 

•  number  of  people  involved  in  some  way 

f  \ 

ULS  systems  will  be  interdependent  webs  of  software-intensive  systems, 

people,  policies,  cultures,  and  economics. 

ULS  systems  are  systems  of  systems  at  internet  scale. 

v _ _ _ J 
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ULS  Systems  and  Systems  of  Systems 


•  Directed:  the  systems  are  integrated  and  built  to  fulfill  specific 
purposes 

•  Acknowledged:  has  recognized  objectives,  a  designated 
manager,  and  resources 

•  Collaborative:  components  voluntarily  agree  to  fulfill  central 
purposes 


•  Virtual:  no  central  authority  or  centrally  agreed  purpose 
. .  .and  beyond 


ULS  Systems 


1  Adapted  from  Maier,  M.  (1998);  "Architecting  Principles  for  Systems-of-Systems";  Systems  Engineering, 
Vol.  1 ,  No.  4  (pp  267-284). 
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Scale  Changes  Everything 


Characteristics  of  ULS  systems  arise  because  of  their  scale. 

•  Decentralization 

•  Inherently  conflicting,  unknowable,  and  diverse  requirements 

•  Continuous  evolution  and  deployment 

•  Heterogeneous,  inconsistent,  and  changing  elements 

•  Erosion  of  the  people/system  boundary 

•  Normal  failures 

•  New  paradigms  for  acquisition  and  policy 


These  characteristics  may  appear  in  today’s  systems  and  systems  of  systems, 
but  in  ULS  systems  they  dominate. 

These  characteristics  undermine  the  assumptions  that  underlie  today’s 
K  software  engineering  approaches. _ 


— ^ — ■  The  Impact  of  Scale 

Software  Engineering  Institute  Carnegie  Mellon  Linda  Northrop:  December  17,  2009 

©  2009  Carnegie  Mellon  University 


Polling  Question  3 


Do  you  see  these  characteristics  in  systems  you  are  working  on? 

1.  Decentralization 

2.  Inherently  conflicting,  unknowable,  and  diverse  requirements 

3.  Continuous  evolution  and  deployment 

4.  Heterogeneous,  inconsistent,  and  changing  elements 

5.  Erosion  of  the  people/system  boundary 
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Today’s  Approaches 


The  Engineering  Perspective  -  for  large  scale  software-intensive  systems 

•  largely  top-down  and  plan-driven 

•  requirements/design/build  cycle  with  standard  well-defined  processes 

•  centrally  controlled  implementation  and  deployment 

•  inherent  validation  and  verification 

The  Agile  Perspective  -  proven  for  smaller  software  projects 

•  fast  cycle/frequent  delivery/test  driven 

•  simple  designs  embracing  future  change  and  refactoring 

•  small  teams  and  retrospective  to  enable  team  learning 

•  tacit  knowledge 


Today’s  approaches  are  based  on  perspectives  that  fundamentally  do  not  cope 
with  the  new  characteristics  arising  from  ultra-large  scale. 

^  The  mentality  oflooking  backward  doesn’t  scale. 
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A  New  Perspective  is  Required 


“The  older  is  not  always  a  reliable  model  for  the  newer,  the 
smaller  for  the  larger,  or  the  simpler  for  the  more 
complex... Making  something  greater  than  any  existing  thing 

necessarily  involves  going  beyond  experience.” 

Henry  Petroski 

Pushing  the  Limits:  New  Adventures  in  Engineering 
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Today  We  Build  “Buildings” 


'Characteristics 
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We  Need  To  Think 


Characteristics 


“Cities  are  places  of  massive 
information  flows,  networks,  and 
conduits,  and  myriad  transitory 
information  exchanges.” 

Howard  Rheinegold:  Smart  Mobs 
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We  Need  to  Think  Ecosystem 


WOODLAND  ECOSYSTEM 


Diverse  users  with  complex  networked 
dependencies  and  intrinsic  adaptive 
behavior 

Has: 

•  Robustness  mechanisms:  achieving 
stability  in  the  presence  of  disruption 

•  Measures  of  health:  diversity, 
population  trends,  other  key  indicators 
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We  Need  to  Think  Socio-Technical 
Ecosystems 


Characteristics 


Socio-technical  ecosystems  include 
people,  organizations,  and  technologies 
at  all  levels  with  significant  and  often 
competing  interdependencies. 

•  There  will  be  competition  for 
resources. 

•  There  will  be  organizations  and 
participants  responsible  for  setting 
policies. 

•  There  will  be  organizations  and 
participants  responsible  for  producing 
ULS  systems. 


•  There  will  need  to  be  local  and  global 
indicators  of  health  that  will  trigger 
necessary  changes  in  policies  and  in 
element  and  system  behavior. 


— ^ — ■  The  Impact  of  Scale 

Software  Engineering  Institute  Carnegie  Mellon  Linda  Northrop:  December  17,  2009 

©  2009  Carnegie  Mellon  University 


Why  a  New  Perspective? 


Characteristics 


There  are  fundamental  assumptions  that  underlie  today’s  software 
engineering  and  software  development  approaches  that  are 
undermined  by  the  characteristics  of  ULS  systems. 

There  are  challenges  associated  with  ULS  systems  that  today’s 
perspectives  are  very  unlikely  to  be  able  to  address. 


r  \ 

In  ULS  systems,  we  now  are  dealing  with  not  just  software  but  an  ecosystem  of 
people,  organizations,  governance,  social  interaction,  hardware,  and  software. 

V _ J 
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Challenges 


ULS  systems  will  present  challenges 
in  three  broad  areas: 

•  Design  and  evolution 

•  Orchestration  and  control 

•  Monitoring  and  assessment 


=  Software  Engineering  Institute 
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Where  Do  We  Focus  Our  Research 


•  Address  the  predominant 
characteristics  of  ULS  systems  and 
the  three  challenge  categories. 

•  Look  for  breakthroughs  not 
incremental  improvement  in  current 
approaches. 

•  Take  a  more  expansive  view  of 
software  research  and  include  its 
interactions  with  associated 
research  in  the  physical  and  social 
sciences. 
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Research  Portfolio 


We  recommend  an  interdisciplinary 
portfolio  of  seven  research  areas 
and  suggested  topics  for 
breakthrough  research  needed  to 
meet  the  challenges  associated  with 
ULS  systems. 

•  Is  not  expressed  in  terms  of 
today’s  “hot”  technologies. 

•  Does  not  supplant  current 
software  research. 

•  Expands  today’s  horizons. 


Software  Engineering  Institute 
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Research  Areas  - 1 


Research  Areas 


6.1*  Human  Interaction:  involves  anthropologists,  sociologists,  and  social  scientists 
conducting  detailed  socio-technical  analyses  of  user  interactions  in  the  field,  with  the 
goal  of  understanding  how  to  construct  and  evolve  such  socio-technical  systems 
effectively. 

6.2  Computational  Emergence:  explores  the  use  of  methods  and  tools  based  on 
economics  and  game  theory  (e.g.,  mechanism  design)  to  ensure  globally  optimal 
ULS  system  behavior  by  exploiting  the  strategic  self  interests  of  the  system’s 
constituencies;  explores  metaheuristics  and  digital  evolution  to  augment  the 
cognitive  limits  of  human  designers. 

6.3  Design:  broadens  the  traditional  technology-centric  definition  of  design  to  include 
people  and  organizations;  social,  cognitive,  and  economic  considerations;  and 
design  structures  such  as  design  rules  and  government  policies. 

6.4  Computational  Engineering:  focuses  on  evolving  the  expressiveness  of 
representations  to  accommodate  the  semantic  diversity  of  many  languages  and 
focuses  on  providing  automated  support  for  computing  the  evolving  behavior  of 
components  and  their  compositions. 

*  Section  in  report  where  research  area  is  described 
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Research  Areas  -  2 


Research  Areas 


•  6.5  Adaptive  System  Infrastructure:  investigates  integrated  development 
environments  and  runtime  platforms  that  will  support  the  decentralized, 
“always-on,”  nature  of  ULS  systems  as  well  as  technologies,  methods,  and 
theories  that  will  enable  ULS  systems  to  be  developed  in  their  deployment 
environments. 

•  6.6  Adaptable  and  Predictable  System  Quality:  focuses  on  how  to 
maintain  quality  in  a  ULS  system  in  the  face  of  continuous  change,  ongoing 
failures,  and  attacks  and  how  to  identify,  predict,  and  control  new  indicators 
of  system  health  (akin  to  the  U.S.  gross  domestic  product)  that  are  needed 
because  of  the  scale  of  ULS  systems. 

•  6.7  Policy,  Acquisition,  and  Management:  focuses  on  transforming 
acquisition  policies  and  processes  to  accommodate  the  rapid  and 
continuous  evolution  of  ULS  systems  by  treating  suppliers  and  supply 
chains  as  intrinsic  and  essential  components  of  a  ULS  system. 
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What  We  Learned  That  We  Want  to  Share 


There  is  an  unstoppable  trend  toward  increasing  scale  in  many  systems 
important  to  our  society. 

Scale  changes  everything. 

Manifestations  of  scale  and  its  attendant  complexity  arise  in  many 
disciplines,  and  can  be  understood  as  a  phenomenon  in  its  own  right. 

New,  interdisciplinary  perspective  and  new  research  in  building  ultra- 
large-scale  systems  is  long  overdue. 


“Since  computation  has  moved  over  the  past  twenty  years  decisively 
closer  to  people,  interfaces  with  social  sciences  such  as  Psychology  and 
Sociology,  besides  Economics,  have  become  increasingly  important” 

Christos  H.  Padadimitriou,  in  “Algorithms,  Games,  and  the  Internet” 

_ ) 
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ULS  Systems  Research  Study  Report 
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Post-Study  Observations 


The  study  does  not  contend  that  all  systems  of  the  future  will  be  ULS 
systems.  Clearly,  they  won’t  be. 

Many  believe  that  there  are  ULS  systems  today. 

What  you  call  a  system  (system  of  systems,  ULS  system,  complex  net- 
centric  system)  is  really  unimportant. 

It  is  important  that  ULS  system  characteristics  are  recognized. 

•  These  characteristics  undermine  the  assumptions  we  make  in  most  current 
technical,  management,  and  acquisition  approaches. 

•  The  ULS  system  perspective  is  helpful  in  understanding  some  of  the  current 
technology  and  management  shortcomings  and  issues  with  system  of 
systems. 

The  research  identified  in  the  ULS  system  study  can  also  have  a 
positive  impact  on  systems  that  are  not  ULS. 
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SEI  Work  on  ULS  Systems 


Research: 

•  Computational  Mechanism  Design 

•  Architecture  in  ULS  system  context 

•  Edge  Programming 

•  Distributed  governance  in  dynamic  contexts 

•  Assurance  in  ULS  systems 

Transition  and  Community  Building: 

•  Website 

•  Technical  Workshops 

•  Continued  publications 

•  Continued  presentations  and  keynotes 
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•  Non-DoD  ULS  system  book  that  includes  essays 


Software  Engineering  Institute  CamegieMellon 


The  Impact  of  Scale 

Linda  Northrop:  December  17,  2009 
©  2009  Carnegie  Mellon  University 


40 


Summing  It  Up 


The  characteristics  of  ULS  systems  undermine  assumptions  of 
traditional  software  and  systems  engineering  practice. 

Approaches  to  meeting  the  challenges  of  ULS  systems  can  be  found 
in,  or  at  least  inspired  by,  other  disciplines,  such  as  biology  and 
economics. 

Many  of  today’s  challenges  are  harbingers  of  future  ULS  system 
challenges,  and  today’s  systems  can  benefit  from  taking  a  “ULS 
system  perspective.” 
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